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DETAILED ACTION 
Reissue Applications 

1 . Applicant is reminded of the continuing obligation under 37 CFR 1 .178(b), to timely 
apprise the Office of any prior or concurrent proceeding in which Patent No. 6,098,082 is or was 
involved. These proceedings would include interferences, reissues, reexaminations, and 
litigation. 

Applicant is further reminded of the continuing obligation under 37 CFR 1.56, to timely 
apprise the Office of any information which is material to patentability of the claims under 
consideration in this reissue application. 

These obligations rest with each individual associated with the filing and prosecution of 
this application for reissue. See also MPEP §§ 1404, 1442.01 and 1442.04. 

Claim Rejections - 35 USC § 112 

2. The following is a quotation of the second paragraph of 35 U.S.C. 1 12: 

The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the 
subject matter which the applicant regards as his invention. 

3. Claims 26 and 27 are rejected under 35 U.S.C. 1 12, second paragraph, as being indefinite 
for failing to particularly point out and distinctly claim the subject matter which applicant 
regards as the invention. Claims 26 and 27 recited the "method of claim 25." Claim 25 has been 
cancelled. For the purposes of the prior art rejection that follows, claims 26 and 27 are 
interpreted as depending from claim 1 . 
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Claim Rejections - 35 USC §103 

1 . The following is a quotation of 35 U.S. C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

2. The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 
(1966), that are applied for establishing a background for determining obviousness under 35 
U.S.C. 103(a) are summarized as follows: 

1 . Determining the scope and contents of the prior art. 

2. Ascertaining the differences between the prior art and the claims at issue. 

3. Resolving the level of ordinary skill in the pertinent art. 

4. Considering objective evidence present in the application indicating obviousness 
or nonobviousness. 

3. Claims 28 and 29 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Behzad Shahraray, and David Gibbon, "Automatic Generation of Pictorial Transcripts of 
Video Programs," February 1995, Proceedings of SPIE 2417 Multimedia Computing and 
Networking 1994 ("Shahraray et al.") in view of U.S. Patent No. 5,699,458 to Sprague. 

4. Regarding claim 28, the Office agrees with Applicant's mapping to Shahraray et al, 
provided in item (7) of the Supplemental Reissue Declaration filed 12 June 2003. Specifically, 
Shahraray et al. disclose "a method for automatically providing a compressed rendition of a 
video program in a format suitable for electronic searching and retrieval, said method comprising 
the steps of: 

a. receiving electronic data representing a condensed version of a video program, 

said video program having a video component and a second information-bearing media 
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component associated therewith, said electronic data representation including a 
representative frame from each segment of the video component of the video program 
and a portion of said second media component associated with said segment (p. 514) ; 

b. automatically transforming said electronic data representation into a hypertext 
format to form a hypertext pictorial transcript {paragraph 4 of p. 515, Fig. 3 on p. 517); 
and 

c. recording said hypertext pictorial transcript in an electronic medium (5th 
paragraph on p. 515). 

While Fig. 3 on page 517 shows an "hypertext" formatted document (HTML), Shahraray et al. 
does not expressly disclose "a hypertext format that includes hypertext links." 
5. Sprague discloses "a hypertext format that includes hypertext links" is known (lines 11- 
14 of column 1: "The World Wide Web (WWW) is a hypertext document network implemented on 
top of the internet. It allows hypertext linking of multimedia documents containing text, sound, 
images, and video... "), and shows "a method for automatically providing a compressed rendition 
of a video program in a format suitable for electronic searching and retrieval" (lines 42-46 of 
column 10: "...a " thumbnail video sequence" may be transmitted to allow browsing of the video 
sequence itself, which comprises a thumbnail version of each intracoded frame within the video 
sequence. This would allow the viewer to efficiently preview the movie before downloading the 
entire sequence. "). Sprague discloses that the user can request the full-quality version of the 
resource of the thumbnail (lines 10-12 of column 9; lines 30-35 of column 8: " This low- 
resolution version of the image can be used when browsing a database of documents containing 
many images in cases of limited transmission bandwidth. When the user identifies an image of 
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interest, the user can request that the full resolution image be transmitted"). One of ordinary 
skill in the art would recognize, from lines 1 1-14 of column 1, that resource requests are made 
using hypertext links. 

6. At the time of the invention, it would have been obvious to a person of ordinary skill in 
the art to present Shahraray's video as Sprague's "thumbnail video sequence" in the hypertext 
document, and to use hypertext links to request the full resolution version of the video. The 
motivation for adding "hypertext links" is to conserve bandwidth, allowing "the viewer to 
efficiently preview the movie before downloading the entire sequence" as suggested by Sprague 
in lines 45-46 of column 10 and lines 30-35 of column 8. Therefore, it would have been obvious 
to combine Shahraray et al. with Sprague to obtain the invention specified in claim 28. 

7. Claim 29 is met by the combination of Shahraray et al. and Sprague, wherein Sprague 
discloses "the step of responding to a query by a user, applied to information stored by the step 
of recording" {browsing and requesting documents as described at lines 10-12 of column 9 and 
lines 30-35 of column 8: " This low-resolution version of the image can be used when browsing a 
database of documents containing many images in cases of limited transmission bandwidth. 
When the user identifies an image of interest, the user can request that the full resolution image 
be transmitted"). The proposed combination as well as the motivation for combining the 
references presented in the rejection of the parent claim apply to this claim and are incorporated 
herein by reference. 

8. Claims 1, 2, 5, 6, 23, and 24 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Behzad Shahraray and David Gibbon, "Automatic Generation of 
Pictorial Transcripts of Video Programs," February 1995, Proceedings of SPIE 2417 
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Multimedia Computing and Networking 1994 ("Shahraray et al.") in view of Tim Berners- 
lee and Robert Cailliau, "The World-Wide Web," 23 September 1992, Computing in High 
Energy Physics 92 (Berners-Lee et al.). 

9. Regarding claim 1, the Office agrees with Applicant's mapping to Shahraray et al, 
provided in item (7) of the Supplemental Reissue Declaration filed 12 June 2003. Specifically, 
Shahraray et al. disclose "a method comprising the steps of: 

d. receiving electronic data representing a condensed version of a video program, 
said video program having a video component and a second information-bearing media 
component associated therewith, said electronic data representation including a 
representative frame from each segment of the video component of the video program 
and a portion of said second media component associated with said segment (p. 514) ; 

e. automatically transforming said electronic data representation into a hypertext 
format to form a hypertext pictorial transcript {paragraph 4 of p. 515, Fig. 3 on p. 517); 
and 

f. recording said hypertext pictorial transcript in an electronic medium (5th 
paragraph on p. 515). 

10. Shahraray et al. disclose "includes a subset of representative frames selected by at least 
one criterion" (section 2: "We use the method of [1] which is based on segmenting the video... A 
single frame is then retained from each of the segments ... "). 

11. While Fig. 3 on page 517 shows an "hypertext" formatted document (HTML), Shahraray 
et al. does not expressly disclose "a hypertext format suitable for electronic searching and 
retrieval" or "responding to a keyword search by a user, performed on said second information- 
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bearing media component stored by said step of recording, where a pictorial transcript that is 
responsive to said keyword search is configured for presentation to said user in [the] form of 
pages, at least some of which are interconnected by hypertext links." 

12. Regarding claim 1, Berners-Lee et al. disclose "a hypertext format suitable for electronic 
searching and retrieval" and "responding to a keyword search by a user, performed on said 
second information-bearing media component stored by said step of recording," (1st paragraph 
of section "Operation ": "The W3 world view is of documents referring to each other by links. 
For its likeness to a spider 's construction, this world is called the Web. This simple view is 
known as the hypertext paradigm. Hypertext alone is not practical when dealing with large sets 
of structured information such as are contained in data bases: adding a search to the hypertext 
model gives W3 its full power (fig. 1). Indexes are special documents which, rather than being 
read, may be searched. To search an index, a reader gives keywords (or other search criteria). "; 
See Figure 1) where a hypertext document "that is responsive to said keyword search is 
configured for presentation to said user in [the] form of pages, at least some of which are 
interconnected by hypertext links" (1st paragraph of section "Operation ": "The result of a 
search is another document containing links to the documents found. "; 2nd paragraph in section 
"Providing Information"] : "An existing server may be taken as an example to be modified and 
enhanced to provide the functionality required. Typically, it is modified to call a program which 
already exists to access the data. The server merely reformats the W3 document address (and/or 
search criteria) into a request to the program, and then reformats the program output as 
hypertext. ") 
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13. At the time of the invention, it would have been obvious to a person of ordinary skill in 
the art to Berners-Lee's "world wide web" paradigm to search and deliver the pictorial hypertext 
documents created by Shahraray. The motivation for doing so would have been to allow "access 
to the universe of online information. . .without regard to where information is, how it is stored, 
or what system is used to manage it." Moreover, such a combination is expressly suggested by 
Shahraray et al. at the second paragraph of section 6. Therefore, it would have been obvious to 
combine Shahraray et al. with Berners-Lee et al. to obtain the invention specified in claim 1 . 

14. Claim 2 is met by the combination of Shahraray et al. and Berners-Lee et al, wherein 
Shaharary et al. disclose "said second media component is closed-caption text" {section 3: "The 
textual information used to generate the pictorial transcript is extracted from the closed caption 
signal... ").The proposed combination as well as the motivation for combining the references 
presented in the rejection of the parent claim apply to this claim and are incorporated herein by 
reference. 

15. Claim 5 is met by the combination of Shahraray et al. and Berners-Lee et al, wherein 
Shahraray et al. disclose "each of said pages includes at least one of said representative frames 
and said second information-bearing media component associated with said at least one 
representative frame" {Figure 3; section 3: "The textual information used to generate the 
pictorial transcript is extracted from the closed caption signal... "). The proposed combination as 
well as the motivation for combining the references presented in the rejection of the parent claim 
apply to this claim and are incorporated herein by reference. 

16. Claim 6 is met by the combination of Shahraray et al. and Berners-Lee et al, wherein 
Shahraray et al. disclose "each of said pages includes not more than a preselected number of said 
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representative frames and said second information-bearing media component associated with the 
representative frames that are in said each of said pages" {Figure 3; Abstract: "The key frames 
and the related text generate a compact multimedia presentation of the contents of the video 
program which lends itself to efficient storage and transmission. " ).The proposed combination as 
well as the motivation for combining the references presented in the rejection of the parent claim 
apply to this claim and are incorporated herein by reference. 

17. Claims 23 and 24 are met by the combination of Shahraray et al. and Berners-Lee et al, 
wherein Shahraray et al. disclose "the step of transmitting said hypertext pictorial transcript over 
a communications network" [claim 23], "said network is the World Wide Web" [claim 24] {2nd 
paragraph of section 6: "A different transcript generator can generate files in the HyperText 
Markup Language (HTML) which can be viewed using one of server HTML viewers.... over the 
Internet World Wide Web. "). The motivation to combine the references given in claim 1 is 
incorporated here by reference. 

18. Claims 3, 7-10, 16, 18-22, 26 and 27 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Behzad Shahraray and David Gibbon, "Automatic Generation of 
Pictorial Transcripts of Video Programs," February 1995, Proceedings of SPIE 2417 
Multimedia Computing and Networking 1994 ("Shahraray et al.") in view of Tim Berners- 
lee and Robert Cailliau, "The World-Wide Web," 23 September 1992, Computing in High 
Energy Physics 92 (Berners-Lee et al.) in view of U.S. Patent No. 5,708,825 to Sotomayor. 

19. Regarding claim 3, Shaharary et al. disclose "said second media component is audio" 
{2nd paragraph of section 1: "Video program are one form of a multimedia presentation 
consisting of at least two media; full motion video, and audio. "); however, Shahrarary et al. does 
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not expressly disclose "wherein said portions of said audio component are represented by 
hypertext anchors." 

20. Sotomayor discloses, portions of an audio component represented by hypertext anchors 
{lines 62-64 of column 6: " A page often has several source anchors 75 with hyperlinks to 
various other pages or to specific locations within pages. "; lines 37-40 of column 1: " These 
hyper linked screen displays can all be of portions of the media data (media data can include, 
e.g., text, graphics, audio, video, etc.) from a single data file...."). 

21 . At the time of the invention, it would have been obvious to a person of ordinary skill in 
the art to provide links to audio as taught by Sotomayor. The motivation for doing so would have 
been to conform to Berners-Lee's paradigm of allowing "access to the universe of online 
information. . .without regard to where information is, how it is stored, or what system is used to 
manage it." Therefore, it would have been obvious to further modify the combination to obtain 
the invention specified in claim 3. 

22. Regarding claim 7, Shahraray et al. does not expressly disclose "wherein said plurality of 
hypertext pages are presented to said user in a scheme selected from a plurality of schemes by 
said user." 

23. Regarding claim 7, Sotomayor discloses "wherein said plurality of hypertext pages are 
presented to said user in a scheme selected from a plurality of schemes by said user" {lines 10-19 
of column 21: "In one embodiment, default summary page templates 154 provide. ..All of these 
design elements can be changed simply by editing the summary page templates 154. "; lines 45- 
50 of column 23: "In one embodiment, when the author runs summary page generator 40, the 
template parser will look in the directory specified as the Template Directory in the Directory 
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dialog box for summary page templates 154 with the filenames listed in the AP.INI file. 
Changing the active template set is a simple matter of specifying a new directory. "). 

24. At the time of the invention, it would have been obvious to a person of ordinary skill in 
the art to use multiple page layouts as taught by Sotomayor to display the pictorial layout 
disclosed by Shahraray et al. The motivation for doing so would have been give the user the 
flexibility to customize the presentation, while maintaining a uniform presentation with less 
effort than manually coding the pages. Therefore, it would have been obvious to combine 
Shahraray et al. with Sotomayor to obtain the invention specified in claim 7. 

25. Regarding claim 8, Shahraray et al. discloses presenting "to said user all of said 
representative frames of said hypertext pictoral transcript" (Figure 3; Abstract: "The key frames 
and the related text generate a compact multimedia presentation of the contents of the video 
program which lends itself to efficient storage and transmission. " ). Shahraray et al. does not 
expressly disclose the said scheme configured to present the pictoral transcript; however, this is 
deemed obvious in view of Sotomayor et al. under the combination proposed in the parent clam. 
The proposed combination as well as the motivation for combining the references presented in 
the rejection of the parent claim apply to this claim and are incorporated herein by reference. 

26. Regarding claim 9, Shahraray et al. discloses presenting "said pictoral representation to 
present to said user a subset of said representative frames, selected by at least one criterion to 
reduce retrieval time of said hypertext pictoral transcript for presentation to said user" (Figure 3; 
Abstract: "The key frames and the related text generate a compact multimedia presentation of the 
contents of the video program which lends itself to efficient storage and transmission. "; see 
section 2). Likewise, Sotomayor further discloses schemes that configure the size of the 
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presentation based on criteria specified by the user {line 66 of column 18 through line 8 of 
column 19: "At start-up time, the author specifies the segment size for pages by selecting a value 
with spinner 124 of the start-up screen... When summary page generator 40 creates summary 
pages 62 and presentation pages 150 at step 47 in FIG. 8, summary page generator 40 divides 
the pages (i.e., output documents 64) that are larger than the segment size into segments no 
larger than the specified size. "). Sotomayor discloses, "a "document" is defined in a broad sense 
to indicate text, pictorial, audio, video and other information stored in one or more computer 
files." See lines 50-53 of column 6. That is, Sotomayor discloses segmenting a "document," see 
e.g. Abstract (d), in the sense of segmenting a video as well. At the time of the invention, it 
would have been obvious to a person of ordinary skill in the art to define a maximum size in 
terms of frames instead of paragraphs. The motivation for doing so would have been use the 
smallest logical unit of video for the segmenting operation. Therefore, it would have been 
obvious to further modify the combination of Sotomayor and Shahraray et al. to obtain the 
invention specified in claim 9. 

27. Claim 10 is met by the combination, wherein Shahraray et al. disclose "said criterion 
removes substantially redundant representative frames" (section 2: "We use the method of [1] 
which is based on segmenting the video... A single frame is then retained from each of the 
segments... "). The proposed combination as well as the motivation for combining the references 
presented in the rejection of the parent claim apply to this claim and are incorporated herein by 
reference. 

28. Regarding claim 16, Shahraray et al. does not use the language "said criterion removes 
representative frames taken from segments below a threshold length"; however, this is deemed 
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inherent to the operation of the content-based sampling method described in section 2. 
Specifically, Shahraray et al. state, "A single frame is the retained from each of the segments." In 
other words, any additional frames from the segment must be removed to meet the criterion of 
one frame per segment. Thus, Shahraray' s criterion removes representative frames taken from 
segments below a threshold length, where the length of the "segment" establishes a threshold 
length. 

Claims 18-22 

29. Regarding claims 18 and 22, Shahraray et al. does not expresly disclose ""a user 
customizable page layout." 

30. Regarding claims 18 and 22, Sotomayor discloses "a user customizable page layout" 
[claim 18] {lines 10-19 of column 21: "In one embodiment, default summary page templates 154 
provide... All of these design elements can be changed simply by editing the summary page 
templates 154. ") and a "plurality of page layouts selectable by a user" [claim 22] {lines 45-50 of 
column 23: "In one embodiment, when the author runs summary page generator 40, the template 
parser will look in the directory specified as the Template Directory in the Directory dialog box 
for summary page templates 154 with the filenames listed in the AP.INI file. Changing the active 
template set is a simple matter of specifying a new directory. "). 

31. At the time of the invention, it would have been obvious to a person of ordinary skill in 
the art to use multiple page layouts as taught by Sotomayor to display the pictorial layout 
disclosed by Shahraray et al. The motivation for doing so would have been give the user the 
flexibility to customize the presentation, while maintaining a uniform presentation with less 
effort than manually coding the pages. 
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32. Regarding claim 19, Shahraray et al. does not expressly disclose "the steps of generating 
and recording an index page." 

33. Regarding claim 19, Sotomayor further discloses "the steps of generating and recording 
an index page (summary pages 80, 140, 100, and 200 described in columns 13 and 14) to the 
hypertext transcript (lines 23-26 of column 15: "Summary page generator 40 also creates a 
hyperlink from each key-topic entry in a summary page to an instance of that key topic in the 
presentation pages 150.... "; see also 250 in Figure 10). 

34. Regarding claim 20, Sotomayor further discloses "said index page (concept summary 
page 200, Figure 10) includes links to individual pages (250) of the hypertext transcript (lines 
28-33 of column 20: "In an alternate embodiment, assuming there are N index entries to be 
hyperlinked to which form a summary page and E is the number of entries that would fit in the 
summary page (E corresponds to the 26 entries for A-to-Z pages), then every N/E index entry 
could be put on a summary page. ") 

35. Regarding claim 21, Sotomayor further discloses "said index page includes hypertext 
index terms (entries for 200 as described in lines 28-33 of column 20; or phrases for 100 as 
showin in Figure 9B and lines 64-67 of column 14) indexed to pages of the hypertext transcript 
(key phrase summary page 100 as described in lines 64-67 of column 14: "Key phrases are then 
assembled in alphabetical order a key-phrase summary page 100 and hyperlinked to the places 
they occur in a presentation page 150. "). 

36. Regarding claims 19-21, it would have been obvious to a person of ordinary skill in the 
art at the time of the invention to the index pages disclosed by Sotomayor to link to portions of 
Shahraray' s pictorial transcript. The motivation for doing so would have been to present an 
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organization that would permit the user to find the relevant portion of the presentation faster than 
the user could by manually navigating through many individual items. Therefore, it would have 
been obvious to combine Sotomayor with Shahraray et al. to obtain the invention specified in 
claims 19-21. 

37. Regarding claim 26, Shahraray et al. does not expressly disclose "said hypertext pages 
are divided based on topic segmentation." 

38. Regarding claim 26, Sotomayor further discloses "said hypertext pages are divided based 
on topic segmentation" {lines 23-26 of column 15: "Summary page generator 40 also creates a 
hyperlink from each key-topic entry in a summary page to an instance of that key_ topic in the 
presentation pages 150.... "). It would have been obvious to a person of ordinary skill in the art at 
the time of the invention to the index pages disclosed by Sotomayor to link to portions of 
Shahraray' s pictorial transcript. The motivation for doing so would have been to present an 
organization that would permit the user to find the relevant portion of the presentation faster than 
manually navigating through many individual items. Therefore, it would have been obvious to 
combine Sotomayor with Shahraray et al. to obtain the invention specified in claim 26. 

39. Regarding claim 27, Shahraray et al. disclose using the "a change in closed-caption 
format" to establish boundaries (last paragraph of section 4: "Therefore, the text segmentation 
boundaries are adjusted by performing lexical analysis and using caption control characters. As 
a result, the text segmentation boundaries are moved to the beginning of sentences. "). 

40. Shahraray et al. does not expressly disclose "said hypertext pages are divided based on a 
change in closed-caption format." 
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41 . Regarding claim 27, Sotomayor discloses "said hypertext pages are divided based on a 
change in" text {changes in topic/concept as described in lines 4-8 of column 15: "In one 
embodiment, as source document 20 is parsed to locate key topics, the text (and other 
multimedia data, if any) is copied into presentation pages 150, and destination anchors are 
inserted into the text corresponding to each key topic placed into the key topic lists for summary 
pages 62. "; Figure 10 shows a hypertext paged divided into a plurality of concept index pages 
250 ). It would have been obvious to a person of ordinary skill in the art at the time of the 
invention to divide Shahraray's hypertext pages according to the analysis disclosed in Sotomayor 
based on a change in the closed-caption format described in section 4 of Shahraray. The 
motivation for doing so would have been to present an organization that would permit the user to 
find the relevant portion of the presentation faster than manually navigating through many 
individual items. Therefore, it would have been obvious to further modify the combination of 
Sotomayor with Shahraray et al. to obtain the invention specified in claim 27. 

42. Claim 11 is rejected under 35 U.S.C. 103(a) as being unpatentable over Behzad 
Shahraray, and David Gibbon, "Automatic Generation of Pictorial Transcripts of Video 
Programs," February 1995, Proceedings of SPIE 2417 Multimedia Computing and 
Networking 1994 ("Shahraray et al.") in view of Tim Berners-lee and Robert Cailliau, 
"The World-Wide Web," 23 September 1992, Computing in High Energy Physics 92 
(Berners-Lee et al.) in view of U.S. Patent No. 5,708,825 to Sotomayor in view of U.S. 
Patent No. 5,699,458 to Sprague. 
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43. Regarding claim 1 1, the combination of Shahraray et al. and Sotomayor does not 
expressly disclose "the step of replacing said substantially redundant frames with hypertext 
anchors." 

44. Sprague discloses "the step of replacing said substantially redundant frames with 
hypertext anchors" (all frames that are not I frames are replaced as disclosed lines 42-46 of 
column 10: "...a " thumbnail video sequence" may be transmitted to allow browsing of the video 
sequence itself which comprises a thumbnail version of each intracoded frame within the video 
sequence. This would allow the viewer to efficiently preview the movie before downloading the 
entire sequence. "). Sprague discloses that the user can request the full-quality version of the 
resource of the thumbnail (lines 10-12 of column 9; lines 30-35 of column 8: " This low- 
resolution version of the image can be used when browsing a database of documents containing 
many images in cases of limited transmission bandwidth. When the user identifies an image of 
interest, the user can request that the full resolution image be transmitted"). 

45. At the time of the invention, it would have been obvious to a person of ordinary skill in 
the art to present Shahraray's video as Sprague's "thumbnail video sequence" in place of the 
redundant, non-I frame video, and to use hypertext links to request the full resolution version of 
the video. The motivation for replacing the redundant frames with "hypertext links" is to 
conserve bandwidth, allowing "the viewer to efficiently preview the movie before downloading 
the entire sequence" as suggested by Sprague in lines 45-46 of column 10 and lines 30-35 of 
column 8. Therefore, it would have been obvious to combine Shahraray et al. with Sprague to 
obtain the invention specified in claim 1 1 . 
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46. Claim 12 is rejected under 35 U.S.C. 103(a) as being unpatentable over Behzad 
Shahraray, and David Gibbon, "Automatic Generation of Pictorial Transcripts of Video 
Programs," February 1995, Proceedings of SPIE 2417 Multimedia Computing and 
Networking 1994 ("Shahraray et al.") in view of Tim Berners-lee and Robert Cailliau, 
"The World-Wide Web," 23 September 1992, Computing in High Energy Physics 92 
(Berners-Lee et al.) in view of U.S. Patent No. 5,708,825 to Sotomayor in view of U.S. 
Patent No. 5,664,227 to Mauldin et al. 

47. Regarding claim 12, the combination of Shahraray et al. and Sotomayor do not disclose 
"said criterion removes alternating ones of sequentially occurring representative frames." 
Mauldin et al. disclose "said criterion removes alternating ones of sequentially occurring 
representative frames" {Figure 3; lines 18-23 of column 7: "The step 235 removes the 
nonrepresentative frames 62 from the series of video frames 60 to create a skimmed video 68 as 
shown in FIG. 4. The skimmed video 68 comprises the representative frames 64a, 64b, 64c, and 
64d. "). 

48. At the time of the invention, it would have been obvious to a person of ordinary skill in 
the art to use Mauldin' s method to remove alternating ones of sequentially occurring 
representative frames. The motivation for doing so would have been to obtain the advantage 
discussed by Mauldin in lines 50-51 of column 3: [achieve] "a reduction of time of up to twenty 
(20) times or more is achieved while retaining most information content." Therefore, it would 
have been obvious to combine Mauldin with Shahraray et al. and Sotomayor to obtain the 
invention specified in claim 12. 
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49. Claims 13 and 14 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Behzad Shahraray, and David Gibbon, "Automatic Generation of Pictorial Transcripts of 
Video Programs," February 1995, Proceedings of SPIE 2417 Multimedia Computing and 
Networking 1994 ("Shahraray et al.") in view of Tim Berners-lee and Robert Cailliau, 
"The World-Wide Web," 23 September 1992, Computing in High Energy Physics 92 
(Berners-Lee et al.) in view of U.S. Patent No. 5,708,825 to Sotomayor in view of U.S. 
Patent No. 5,764,235 to Hunt et al. 

50. Regarding claims 13 and 14, the combination of Shahraray et al. and Sotomayor disclose 
removing frames based on a criterion as recited in the parent claim. The combination does not 
expressly disclose "said criterion removes representative frames below a prescribed image size" 
[claim 13], or "said criterion removes representative frames above a prescribed image size" 
[claim 14]. Hunt et al. discloses removing image below or above a prescribed image size from 
HTML pages and replacing them with a size better suited to the quality and bandwidth 
requirements {lines 49-55 of column 5: "Then, the web browser 204 searches through the web 
page HTML page to determine whether or not graphical image files are contained within the 
web page... The web server 202, upon receiving the request for the image file, forwards the 
appropriate image file to the web browser 204 through the Internet 206 and the links 208 and 
210. "; lines 40-42 of column 11: "If not, the file size for the determined image file is set 1108 to 
user_size, which indicates that the file size is set by a user's choice or expected choice. "; lines 
59-64 of column 11: "Thus, blocks 1114 and 1116 combine to limit the file size to the 

server _size, which is the maximum file size that the web server is willing to support. As an 
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example, if the web server is experiencing a heavy load, the web server can reduce the amount of 
data it needs to transmit to requesting web browsers by lowering the server _size. "). 
51. At the time of the invention, it would have been obvious to a person of ordinary skill in 
the art to incorporate a criterion removes representative frames below/above a prescribed image 
size in the HTML pictorial transcripts generated by Shahraray et al. The motivation for doing so 
would have been to enable Shahraray's pictorial transcript to be transmitted more flexibly and 
efficiently as suggested by Hunt et al. in lines 20-27: 

As a result, the amount of data transmitted is customized for the particular situation. 

Hence, excess data need not be transmitted when the requester does not need or desire it. 

Alternatively, a request for a very high quality image can be satisfied. Accordingly, the 

invention makes significantly better and more intelligent use of the available bandwidth 

of the network environment. 
Therefore, it would have been obvious to combine Hunt et al. with Shahraray et al. and 
Sotomayor to obtain the invention specified in claims 13 and 14. 

53. Claim 15 is rejected under 35 U.S.C. 103(a) as being unpatentable over Behzad 
Shahraray, and David Gibbon, "Automatic Generation of Pictorial Transcripts of Video 
Programs," February 1995, Proceedings of SPIE 2417 Multimedia Computing and 
Networking 1994 ("Shahraray et al.") in view of Tim Berners-lee and Robert Cailliau, 
"The World-Wide Web," 23 September 1992, Computing in High Energy Physics 92 
(Berners-Lee et al.) in view of U.S. Patent No. 5,708,825 to Sotomayor in view of Behzad 
Shahraray, "Scene Change Detection and Content-Based Sampling of Video Sequences," 
1995, SPIE 2419, pp. 2-13. 
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54. Regarding claim 15, Shahraray et al. disclose, "the generator module can utilize 
information regarding repetitive occurrence of identical (or similar) representative images (e.g., 
when the video switches between two scenes) to prevent repeated occurrences of these images in 
the printed version" in the first paragraph of section 6; however, the combination of Shahraray et 
al. and Sotomayor does not expressly disclose "wherein said criterion removes representative 
frames that differ from other representative frames by less than a prescribed amount." 

55. Shahraray discloses detecting shot boundaries by identifying "frames that differ from 
other representative frames by less than a prescribed amount" (2nd paragraph of section 2.1: 
"The match between the two images referred to as the Image Match (IM), and is defined as.... "; 
last paragraph of section 2.1: "The IM signal computed this way is small when computed over 
images belonging to the same shot, and shows pronounced increase when applied to images 
from different shots. Therefore, it can be used to detect scene cuts by a thresholding process as it 
has been done previously.... "). 

56. At the time of the invention, it would have been obvious to a person of ordinary skill in 
the art to use the method disclosed in the Shahraray reference to detect scene changes in the 
pictorial transcript creation method disclosed by the combination of Shahraray et al. and 
Sotomayor. The motivation for doing so would have been to summarize the pictorial transcript 
by using frames from different scenes "to prevent repeated occurrences of these images" as 
suggested by Shahraray et al. in the first paragraph of section 6. Therefore, it would have been 
obvious to combine Shahraray with the combination of Shahraray et al. and Sotomayor to obtain 
the invention specified in claim 15. 
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57. Claim 17 is rejected under 35 U.S.C. 103(a) as being unpatentable over Behzad 
Shahraray, and David Gibbon, "Automatic Generation of Pictorial Transcripts of Video 
Programs," February 1995, Proceedings of SPIE 2417 Multimedia Computing and 
Networking 1994 ("Shahraray et al.") in view of Tim Berners-lee and Robert Cailliau, 
"The World-Wide Web," 23 September 1992, Computing in High Energy Physics 92 
(Berners-Lee et al.) in view of U.S. Patent No. 5,708,825 to Sotomayor in view of U.S. 
Patent No. 5,692,093 to Iggulden et al. 

58. Regarding claim 17, the combination of Shahraray et al. and Sotomayor do not expressly 
disclose "said criterion removes representative frames taken from advertisements." 

59. Iggulden et al. teaches generally that "representative frames taken from advertisements" 
can be removed from a video signal to the benefit of a viewer during playback {lines 60-67 of 
column 3: " The video signal is divided into segments defined by transitions from or to black 
frames and silent frames and each segment is then analyzed with respect to surrounding 
segments to determine whether it is part of a commercial message or program material. 
Contiguous groups of segments that are classified as commercial messages define a commercial 
cluster that will be scanned past during playback of the recorded videotape. ") 

60. At the time of the invention, it would have been obvious to a person of ordinary skill in 
the art to remove commercials as taught by Iggulden et al. from Shahraray's pictorial transcript 
because the commercial frames would not have been representative of content the video 
program, and thus, would not contribute to the usefulness of the pictorial transcript. Therefore, it 
would have been obvious to combine Shahraray et al. and Sotomayor with Shahraray et al. and 
Sotomayor with the teachings of Iggulden et al. to obtain the invention specified in claim 17. 
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Response to Arguments 

61 . Applicant's arguments filed 14 July 2009 have been fully considered but they are not 
persuasive. 

62. Regarding claim 28, in response to applicant's argument that the references fail to show 
certain features of applicant's invention, it is noted that the features upon which applicant relies 
(i.e., "hyperlink text embedded in the thumbnail video sequence") are not recited in the rejected 
claim(s). Although the claims are interpreted in light of the specification, limitations from the 
specification are not read into the claims. See In re Van Geuns, 988 F.2d 1 181, 26 

USPQ2d 1057 (Fed. Cir. 1993). Applicant's claims require that "a hypertext format that includes 
hypertext links," which is met by Sprague's hypertext format that allows hypertext linking of 
multimedia documents described at lines 1 1-13 of column 1. That is, Applicant's claims are 
directed to the format itself. Moreover, it should be noted that Sprague teaches that a user can 
request data by selecting an icon representing an image that links to the data. See, e.g. , Sprague 
at col. 8, 1. 57-65. For at least this reason, Applicant's arguments directed to claim 28 are not 
persuasive. 

63. Regarding the arguments directed to the newly amended claims, Applicant's attention is 
directed to the rejection presented in this action. 

64. Regarding claim 18, Applicant argues that there is no teaching or suggestion that the user 
has the ability to customize the page layout because the reference refers to the author of the 
document before its storage. In response, it is submitted that Applicant's distinction does not 
appear to be reflected in the claims because the claims explicitly recite "a user customizable 
layout," and say nothing about "the entity (typically individual) that performs the query" as 
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argued by Applicant. Moreover, Applicant's interpretation does not appear to be supported by 
the descriptive portion of the specification. Should Applicant present this argument in future 
responses, Applicant is encouraged to support this argument with specific reference to the 
portion of the specification that discusses this feature. 

65. Regarding claim 11, Applicant argues that the Sprague reference does not teach 
"replacement of substantially redundant frames." However, Sprague discloses all frames that are 
not I frames are replaced as disclosed lines 42-46 of column 10, which is analogous to the 
"replacement of substantially redundant frames" because the method replaces the redundant 
frames in the full version with a smaller subset of frames the in the delivered version. For at least 
this reason, Applicant's argument is not persuasive. 

66. Regarding claim 12, Applicant argues that the Mauldin et al. reference teaches away from 
the recited claim by discussing removing non-representative frames. However, Mauldin's non- 
representative frames are removed from the set of representative frames and are called "non- 
representative" because they are removed. The only difference being that Applicant's chooses to 
name the removed frames "representative." Mauldin's method essentially differs from 
Applicant's claimed invention in vocabulary alone. For at least this reason, Applicant's argument 
is not persuasive. 

67. Regarding claims 13 and 14, Applicant argues against the references individually. One 
cannot show nonobviousness by attacking references individually where the rejections are based 
on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In 
re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). The test for obviousness is not 
whether the features of a secondary reference may be bodily incorporated into the structure of 
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the primary reference; nor is it that the claimed invention must be expressly suggested in any one 
or all of the references. Rather, the test is what the combined teachings of the references would 
have suggested to those of ordinary skill in the art. See In re Keller, 642 F.2d 413, 208 USPQ 
871 (CCPA 1981). In this case, Hunt et al. discloses removing image below or above a 
prescribed image size from HTML pages and replacing them with a size better suited to the 
quality and bandwidth requirements. At the time of the invention, it would have been obvious to 
a person of ordinary skill in the art to incorporate a criterion removes representative frames 
below/above a prescribed image size in the HTML pictorial transcripts generated by Shahraray et 
al. The motivation for doing so would have been to enable Shahraray's pictorial transcript to be 
transmitted more flexibly and efficiently. For at least this reason, Applicant's argument is not 
persuasive. 

68. Regarding claim 15, Applicant argues that the notation of detecting shot boundaries does 
not teach or suggest removing representative frames that differ from other representative frames 
by less than a prescribed amount. However, the Shahraray reference expressly discloses "match 
between the two images referred to as the Image Match (IM)", which "pronounced increase 
when applied to images from different shots" (section 2.1). Thus, the teachings are directed to 
detecting "different shots" and not shots where "the representative frames" are "extremely 
similar" as Applicant argues. Regarding the assertion that the Shahraray method detects shot 
boundaries where "the representative frames" are "extremely similar," it is unclear how 
Applicant arrives at this conclusion because this statement is not accompanied by a citation and 
appears to be unsupported by in the Shahraray reference. For at least this reason, Applicant's 
argument is not persuasive. 
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69. Regarding claim 17, Applicant argues "scanning past presented information is different 
from removing representative frames." This is incorrect because Igguden's teaching of "scanned 
past during playback of the recorded videotape" removes those frames from the video signal sent 
to the display. For at least this reason, Applicant's argument is not persuasive. 

Conclusion 

70. Applicant's amendment necessitated the new ground(s) of rejection presented in this 
Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). 
Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1 .136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the date of this 
final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to JASON M. REPKO whose telephone number is (571)272-8624. 
The examiner can normally be reached on Monday through Friday 8:00 am - 4:30 pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Ulka Chauhan can be reached on (571)272-7782. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 



/Jason M Repko/ 
Examiner, Art Unit 2628 



